{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HTML解析入门及准备URL生成连续技\n",
    "![for humans](https://requests-html.kennethreitz.org/_static/requests-html-logo.png#thumbnail)\n",
    "\n",
    "*  本周主要内容：批量抓取页面基础及技巧\n",
    "*  上周主要内容：HTML解析（parse HTML）及准备URL生成连续技\n",
    "*  20春_Web数据挖掘_week04\n",
    "*  电子讲义设计者：廖汉腾, 许智超\n",
    "<br/>\n",
    "<br/>\n",
    "\n",
    "-----\n",
    "## 复习\n",
    "\n",
    "复习：上周内容，实践\n",
    "\n",
    "* 猎聘PC版 liepin.com 取工作URL参数的牛肉\n",
    "* 如何生成一连串新URL以进一步爬取数据\n",
    "\n",
    "\n",
    "-----\n",
    "## 本周内容及学习目标\n",
    "\n",
    "本周内容聚焦在\n",
    "\n",
    "<mark> 如何有系统的把更多页数据(相同结构)作系统性爬取 </mark>  \n",
    "\n",
    "<mark>（然) </mark>\n",
    "系统地爬取更多页数据，需要考虑如果爬取出现中断，是否可以做代码设计/调整，使其可以从中断的地方开始，而不需要从头开始，\n",
    "这时就需要做好数据的备份和存储。（考虑迭代）\n",
    "\n",
    "为此，我们需要学习\n",
    "\n",
    "* 翻页：参数字典的拆解\n",
    "  * xpath\n",
    "  * 建构参数模板\n",
    "  * 建构参数字典\n",
    "* 翻页：系统性迭代\n",
    "  * robots.txt\n",
    "  * 频率及时间\n",
    "* 翻页：数据备份与整合\n",
    "  * 储存备份\n",
    "  * 数据整合\n",
    "  \n",
    "### 目标\n",
    "1. 使用 requests-html 爬取并存取网页文字档，查找[requests-html 中文文档](https://cncert.github.io/requests-html-doc-cn/#/)\n",
    "2. 熟悉 [xpath 语法](https://www.w3cschool.cn/xpath/xpath-syntax.html)丶[xpath 节点](https://www.w3cschool.cn/xpath/xpath-nodes.html)\n",
    "3. 使用 [xpath cheatsheet](https://devhints.io/xpath)\n",
    "  * 在 Chrome Inspector 使用\n",
    "  * 在 requests-html (Python) 使用\n",
    "4. 简易使用 [pd.DataFrame](https://www.pypandas.cn/doc/getting_started/dsintro.html#dataframe)\n",
    "5. 参数字典的拆解与迭代\n",
    "6. 翻页数据备份与整合"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "/* 本电子讲义使用之CSS */\n",
       "div.code_cell {\n",
       "    background-color: #e5f1fe;\n",
       "}\n",
       "div.cell.selected {\n",
       "    background-color: #effee2;\n",
       "    font-size: 2rem;\n",
       "    line-height: 2.4rem;\n",
       "}\n",
       "div.cell.selected .rendered_html table {\n",
       "    font-size: 2rem !important;\n",
       "    line-height: 2.4rem !important;\n",
       "}\n",
       ".rendered_html pre code {\n",
       "    background-color: #C4E4ff;   \n",
       "    padding: 2px 25px;\n",
       "}\n",
       ".rendered_html pre {\n",
       "    background-color: #99c9ff;\n",
       "}\n",
       "div.code_cell .CodeMirror {\n",
       "    font-size: 2rem !important;\n",
       "    line-height: 2.4rem !important;\n",
       "}\n",
       ".rendered_html img, .rendered_html svg {\n",
       "    max-width: 60%;\n",
       "    height: auto;\n",
       "    float: right;\n",
       "}\n",
       "\n",
       ".rendered_html img[src*=\"#full\"], .rendered_html svg[src*=\"#full\"] {\n",
       "    max-width: 100%;\n",
       "    height: auto;\n",
       "    float: none;\n",
       "}\n",
       "\n",
       ".rendered_html img[src*=\"#thumbnail\"], .rendered_html svg[src*=\"#thumbnail\"] {\n",
       "    max-width: 15%;\n",
       "    height: auto;\n",
       "}\n",
       "\n",
       "/* Gradient transparent - color - transparent */\n",
       "hr {\n",
       "    border: 0;\n",
       "    border-bottom: 1px dashed #ccc;\n",
       "}\n",
       ".emoticon{\n",
       "    font-size: 5rem;\n",
       "    line-height: 4.4rem;\n",
       "    text-align: center;\n",
       "    vertical-align: middle;\n",
       "}\n",
       ".bg-split_apply_comine {\n",
       "    width: 500px;     \n",
       "    height: 300px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -10px -10px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-comine {\n",
       "    width: 175px;\n",
       "    height: 150px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -280px -80px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-apply {\n",
       "    width: 155px;\n",
       "    height: 225px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -160px -30px;\n",
       "    float: right;\n",
       "}\n",
       ".bg-split {\n",
       "    width: 205px;\n",
       "    height: 225px;\n",
       "    background: url('02_split-apply-comine_500x300.png') -10px -30px;\n",
       "    float: right;\n",
       "}\n",
       ".break {\n",
       "                   page-break-after: right; \n",
       "                   width:700px;\n",
       "                   clear:both;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>\n",
    "/* 本电子讲义使用之CSS */\n",
    "div.code_cell {\n",
    "    background-color: #e5f1fe;\n",
    "}\n",
    "div.cell.selected {\n",
    "    background-color: #effee2;\n",
    "    font-size: 2rem;\n",
    "    line-height: 2.4rem;\n",
    "}\n",
    "div.cell.selected .rendered_html table {\n",
    "    font-size: 2rem !important;\n",
    "    line-height: 2.4rem !important;\n",
    "}\n",
    ".rendered_html pre code {\n",
    "    background-color: #C4E4ff;   \n",
    "    padding: 2px 25px;\n",
    "}\n",
    ".rendered_html pre {\n",
    "    background-color: #99c9ff;\n",
    "}\n",
    "div.code_cell .CodeMirror {\n",
    "    font-size: 2rem !important;\n",
    "    line-height: 2.4rem !important;\n",
    "}\n",
    ".rendered_html img, .rendered_html svg {\n",
    "    max-width: 60%;\n",
    "    height: auto;\n",
    "    float: right;\n",
    "}\n",
    "\n",
    ".rendered_html img[src*=\"#full\"], .rendered_html svg[src*=\"#full\"] {\n",
    "    max-width: 100%;\n",
    "    height: auto;\n",
    "    float: none;\n",
    "}\n",
    "\n",
    ".rendered_html img[src*=\"#thumbnail\"], .rendered_html svg[src*=\"#thumbnail\"] {\n",
    "    max-width: 15%;\n",
    "    height: auto;\n",
    "}\n",
    "\n",
    "/* Gradient transparent - color - transparent */\n",
    "hr {\n",
    "    border: 0;\n",
    "    border-bottom: 1px dashed #ccc;\n",
    "}\n",
    ".emoticon{\n",
    "    font-size: 5rem;\n",
    "    line-height: 4.4rem;\n",
    "    text-align: center;\n",
    "    vertical-align: middle;\n",
    "}\n",
    ".bg-split_apply_comine {\n",
    "    width: 500px;     \n",
    "    height: 300px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -10px -10px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-comine {\n",
    "    width: 175px;\n",
    "    height: 150px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -280px -80px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-apply {\n",
    "    width: 155px;\n",
    "    height: 225px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -160px -30px;\n",
    "    float: right;\n",
    "}\n",
    ".bg-split {\n",
    "    width: 205px;\n",
    "    height: 225px;\n",
    "    background: url('02_split-apply-comine_500x300.png') -10px -30px;\n",
    "    float: right;\n",
    "}\n",
    ".break {\n",
    "                   page-break-after: right; \n",
    "                   width:700px;\n",
    "                   clear:both;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 基本模块\n",
    "import pandas as pd   ## （然） pandas是专门为处理表格和混杂数据设计的，df：任意的Pandas DataFrame对象\n",
    "from requests_html import HTMLSession  \n",
    "\n",
    "## （然）1. 建立Session：\n",
    "## from requests_html import HTMLSession\n",
    "## session = HTMLSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0. 上周整合代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "edu         7\n",
      "经验         11\n",
      "薪水         92\n",
      "时间         21\n",
      "职称        187\n",
      "公司地点       62\n",
      "公司名称       62\n",
      "链结        196\n",
      "公司URL      63\n",
      "热门公司类型      6\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>职称</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>公司名称</th>\n",
       "      <th>edu</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>天能电池集团股份有限公司</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>华为</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>恒大集团</th>\n",
       "      <th>硕士及以上</th>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>奥比中光</th>\n",
       "      <th>硕士及以上</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>天能电池集团股份有限公司</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>广日股份</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>敏实集团</th>\n",
       "      <th>大专及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>新城控股集团住宅开发事业部</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>明略科技集团</th>\n",
       "      <th>统招本科</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>风范股份</th>\n",
       "      <th>本科及以上</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>93 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     职称\n",
       "公司名称          edu      \n",
       "天能电池集团股份有限公司  统招本科   18\n",
       "华为            本科及以上  14\n",
       "恒大集团          硕士及以上  10\n",
       "奥比中光          硕士及以上   9\n",
       "天能电池集团股份有限公司  本科及以上   9\n",
       "...                  ..\n",
       "广日股份          本科及以上   1\n",
       "敏实集团          大专及以上   1\n",
       "新城控股集团住宅开发事业部 本科及以上   1\n",
       "明略科技集团        统招本科    1\n",
       "风范股份          本科及以上   1\n",
       "\n",
       "[93 rows x 1 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 上周C-1B-5 建构 参数模板  \n",
    "参数_compTag_用户体验 = {'中国500强': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['155'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, '2018互联网300强': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['182'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, '制造业500强': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['186'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, 'AI创新成长50强 ': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['189'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, '独角兽': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['130'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}, '上市公司': {'init': ['-1'], 'headckid': ['58d828c357a8cb19'], 'flushckid': ['1'], 'fromSearchBtn': ['2'], 'keyword': ['用户体验'], 'compTag': ['156'], 'ckid': ['58d828c357a8cb19'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['6aa779111c1b4ca77cff3648d9dee049'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['6aa779111c1b4ca77cff3648d9dee049']}}\n",
    "\n",
    "## （然）\n",
    "# init，headckid，flushckid等是点击“中国500强”后链接里面的内容，拆分url后一个个对应\n",
    "\n",
    "##（然）\n",
    "# * comTag 是不同的公司选择器, 数值不样, 对映到不同类型的公司\n",
    "# * keyword 是搜查关键字\n",
    "\n",
    "# 上周C-1   多个页面准备测试1 中国500强\n",
    "url = \"https://www.liepin.com/zhaopin/\"  ### （然）将猎聘招聘网的网址赋值给了url\n",
    "session = HTMLSession()                  ### （然）建立session\n",
    "payload = 参数_compTag_用户体验['中国500强']  ### （然）payload 字面意思 “有效载荷，有效负荷，有效载重”\n",
    "r = session.get( url, params = payload)   ### （然）获取数据\n",
    "\n",
    "# r.url\n",
    "\n",
    "# 上周C-2  简化 A-1   单一页面爬+解析\n",
    "session = HTMLSession()  ## （然）解析数据\n",
    "\n",
    "def requests_liepin( url, params):\n",
    "    r = session.get( url , params = payload)  ## （然）获取数据内容\n",
    "\n",
    "    # 先取特定元素, 精准打击其子后辈\n",
    "    主要元素 = r.html.xpath( '//ul[@class=\"sojob-list\"]/li')  ## 找到特定元素的xpath\n",
    "\n",
    "    # 作为xpath字典，键为我要抓的牛肉名称，值为xpath\n",
    "    dict_xpaths={ \n",
    "        'text': {\n",
    "            'edu':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]',   ##  （然）找到学历的xpath\n",
    "            '经验':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]/following-sibling::span',   ## （然）找到经验的xpath\n",
    "            '薪水':    '//div[contains(@class,\"job-info\")]/p/span[@class=\"text-warning\"]',  ## （然）找到薪水的xpath\n",
    "            '时间':    '//div[contains(@class,\"job-info\")]/p/time/@title',   ## （然）找到时间的xpath\n",
    "            '职称':    '//div[contains(@class,\"job-info\")]/h3/a',    ## （然）找到职位名称的xpath\n",
    "            '公司地点': '//div[contains(@class,\"job-info\")]/p/a',   ## （然）找到公司地点的xpath\n",
    "            '公司名称': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a', ## （然）找到公司名称的xpath\n",
    "        },\n",
    "        'text_content': {\n",
    "        },\n",
    "        'href': {\n",
    "            '链结':    '//div[contains(@class,\"job-info\")]/h3/a',   ## （然）找到链结的xpath\n",
    "            '公司URL': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a', ## （然）找到公司url的xpath\n",
    "        }\n",
    "    }\n",
    "\n",
    "    def get_e_text_content(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [e.xpath(_xpath_)[0].lxml.text_content() for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_text(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [\"\".join([x.strip() if type(x) is str else x.text.strip() for x in e.xpath(_xpath_)]) for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_href(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [list(e.xpath(_xpath_, first=True).absolute_links)[0] \\     ### （然）只取第一个来做列表 \n",
    "                   if len(e.xpath(_xpath_, first=True).absolute_links) >= 1  \\\n",
    "                   else \"\" for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    # 只对主要元素下进行.xpath取值\n",
    "    数据字典 = dict()\n",
    "\n",
    "    数据字典 = {k:get_e_text_content(v) for k,v in dict_xpaths['text_content'].items()}  ##（然）以下三行都是将上方的列表for循环遍历抓取内容存入新的字典当中？\n",
    "    数据字典.update({k:get_e_text(v) for k,v in dict_xpaths['text'].items()})\n",
    "    数据字典.update({k:get_e_href(v) for k,v in dict_xpaths['href'].items()})\n",
    "\n",
    "    数据 = pd.DataFrame(数据字典)\n",
    "    #数据.to_excel(\"20春_Web数据挖掘_week03_liepin.xlsx\", sheet_name=\"搜查结果\")\n",
    "    return (数据)\n",
    "\n",
    "\n",
    "# 上周C-3   多个页面\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "\n",
    "list_df = list()\n",
    "for k,v in 参数_compTag_用户体验.items():\n",
    "    payload = v\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    df = df.assign (热门公司类型 = k)    \n",
    "    list_df.append(df)\n",
    "\n",
    "df_all = pd.concat(list_df)\n",
    "df_all\n",
    "\n",
    "# 上周C-4   输出\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week03_liepin_各热门公司类型.xlsx\", sheet_name=\"搜查结果\") ##（然）将数据存入表格\n",
    "\n",
    "# 上周C-5 Pandas  基本能力\n",
    "\n",
    "print (df_all.nunique())\n",
    "df_all[['edu']].drop_duplicates()\n",
    "\n",
    "df_all.groupby(['公司名称','edu']).agg({\"职称\":\"count\"}).sort_values(by='职称', ascending=False) ##（然）使数据相对应输出？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "-----\n",
    "\n",
    "## 本周实践目标\n",
    "<mark> 如何有系统的把更多页数据(相同结构)作系统性爬取 </mark>[猎聘PC版](https://www.liepin.com/zhaopin/)\n",
    "* 翻页：参数字典的拆解\n",
    "  * xpath解析翻页a/@href\n",
    "  * 建构参数模板\n",
    "  * 建构参数字典\n",
    "* 翻页：系统性迭代\n",
    "  * robots.txt\n",
    "  * 频率及时间\n",
    "* 翻页：数据备份与整合\n",
    "  * 储存备份\n",
    "  * 数据整合"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：参数字典的拆解\n",
    "## xpath解析翻页a/@href\n",
    "\n",
    "#### （然） 获取下一页按钮的链接\n",
    " html.xpath(‘//a[text()=\"下一页\"]/@href‘)      ## 可以实现翻页"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A-0   单一页面\n",
    "import requests_html   ## （然）引用requests模块\n",
    "from requests_html import HTMLSession  ## 引用设立session值\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/?keyword=PRD\"   ### （然）将页面链接赋值给url\n",
    "session = HTMLSession()                          ### （然）设立session\n",
    "r = session.get( url )  ###（然） 获取页面信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element 'a' class=('first', 'disabled') href='javascript:;' title='首页'>, <Element 'a' class=('disabled',) href='javascript:;'>, <Element 'a' class=('current',) href='javascript:;'>, <Element 'a' href='/zhaopin/?init=-1&headckid=b17bd0eb181d2ac5&fromSearchBtn=2&keyword=PRD&ckid=b17bd0eb181d2ac5°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=a63f404e1c48764af8e707c0cebc602c&d_curPage=0&d_pageSize=40&d_headId=a63f404e1c48764af8e707c0cebc602c&curPage=1'>, <Element 'a' href='/zhaopin/?init=-1&headckid=b17bd0eb181d2ac5&fromSearchBtn=2&keyword=PRD&ckid=b17bd0eb181d2ac5°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=a63f404e1c48764af8e707c0cebc602c&d_curPage=0&d_pageSize=40&d_headId=a63f404e1c48764af8e707c0cebc602c&curPage=2'>, <Element 'a' href='/zhaopin/?init=-1&headckid=b17bd0eb181d2ac5&fromSearchBtn=2&keyword=PRD&ckid=b17bd0eb181d2ac5°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=a63f404e1c48764af8e707c0cebc602c&d_curPage=0&d_pageSize=40&d_headId=a63f404e1c48764af8e707c0cebc602c&curPage=3'>, <Element 'a' href='/zhaopin/?init=-1&headckid=b17bd0eb181d2ac5&fromSearchBtn=2&keyword=PRD&ckid=b17bd0eb181d2ac5°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=a63f404e1c48764af8e707c0cebc602c&d_curPage=0&d_pageSize=40&d_headId=a63f404e1c48764af8e707c0cebc602c&curPage=4'>, <Element 'a' href='/zhaopin/?init=-1&headckid=b17bd0eb181d2ac5&fromSearchBtn=2&keyword=PRD&ckid=b17bd0eb181d2ac5°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=a63f404e1c48764af8e707c0cebc602c&d_curPage=0&d_pageSize=40&d_headId=a63f404e1c48764af8e707c0cebc602c&curPage=1'>, <Element 'a' class=('last',) href='/zhaopin/?init=-1&headckid=b17bd0eb181d2ac5&fromSearchBtn=2&keyword=PRD&ckid=b17bd0eb181d2ac5°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=a63f404e1c48764af8e707c0cebc602c&d_curPage=0&d_pageSize=40&d_headId=a63f404e1c48764af8e707c0cebc602c&curPage=9' title='末页'>]\n"
     ]
    }
   ],
   "source": [
    "## 代码实验一：\n",
    "## （然）先看一个翻页，看链接\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a'## （然）找到翻页的xpath\n",
    "print (r.html.xpath(xpath_翻页a))         ## （然）将其打印，可以发现获取的结果链接都十分得长"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element 'a' href='/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=1'>, <Element 'a' href='/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=2'>, <Element 'a' href='/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=3'>, <Element 'a' href='/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=4'>, <Element 'a' href='/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=1'>, <Element 'a' class=('last',) href='/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=9' title='末页'>]\n",
      "{'2': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=1', '3': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=2', '4': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=3', '5': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=4', '下一页': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=1', '': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=9'}\n"
     ]
    }
   ],
   "source": [
    "# A-1  xpath 解析翻页a/@href\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a' # 有disabled, current等href是javascript   ## （然）先找到翻页的xpath\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'     ## (然) 判别式保留，@href是属性，以/zhaopin结尾的都要\n",
    "print (r.html.xpath(xpath_翻页a)) # 物件   ## （然）将获取的结果打印出来\n",
    "\n",
    "href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]  ## （然）获取第一个列表元素，设置了for循环\n",
    "#print (href_列表)\n",
    "\n",
    "文字_列表 = [x.text for x in r.html.xpath(xpath_翻页a)]\n",
    "#print (文字_列表)\n",
    "\n",
    "href_字典 = {x.text:x.xpath('//@href')[0]  for x in r.html.xpath(xpath_翻页a)}\n",
    "print (href_字典)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 观察：\n",
    "此网页是否给出开始丶步进丶及结束的信息，以方便我们完成迭代设置\n",
    "\n",
    "<mark>（然）</mark>\n",
    "- 迭代是重复反馈过程的活动，其目的通常是为了逼近所需目标或结果。每一次对过程的重复称为一次“迭代”，而每一次迭代得到的结果会作为下一次迭代的初始值。\n",
    "- 网页给出了开始，结束的信息，@href是属性，以/zhaopin结尾的都要，每条链接结尾的curpage代表的是页数，如curpage=1代表的是第一页，以此类推\n",
    "------------------------------------------------------------------------------------------------------------------------------------\n",
    "\n",
    "* 老问题 URL太长，用上周的URL+query参数解析与pandas数据框找到异同之处\n",
    "\n",
    "<mark>（然）</mark>\n",
    "- url很长，但是可以拆分\n",
    "--------------------------------------------------------------------------------------------------------------------------------------\n",
    "\n",
    "* 老问题 怎麽系统化出URL？用上周的URL+query参数解析与pandas数据框找到异同之处的时候，顺便构建参数字典，至少让以下参数可调\n",
    "  * 搜索关键词：上周keyword\n",
    "  * 页码在哪？  （然）页码在链接的结尾，curpage是页码\n",
    "* 实践挑战：如何把上周代码模块化为我们所用？\n",
    "（然）通过建构参数模板curpage？循环遍历？\n",
    "-----\n",
    "\n",
    "## 建构参数模板\n",
    "\n",
    "```python\n",
    "\n",
    "# 上周B-1 使用 urllib.parse 解析\n",
    "from urllib.parse import urlparse, parse_qs\n",
    "\n",
    "\n",
    "# 上周B-2 使用 pd.DataFrame进行 unuinque()相异值计量比对 \n",
    "import pandas as pd\n",
    "df = pd.DataFrame([ urlparse(x) for x in 公司数据选择器链结.values()])\n",
    "print(df.nunique())\n",
    "\n",
    "# 上周B-3 针对query 再解析之 \n",
    "#df_qs = pd.DataFrame([ parse_qs(x) for x in df['query'] ])\n",
    "df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "print(df.nunique())\n",
    "\n",
    "# 上周B-4 建构 参数模板 及 字典_compTag\n",
    "def parse_url_qs_for_compTag (url):\n",
    "    six_parts = urlparse(url) \n",
    "    out = parse_qs(six_parts.query)\n",
    "    return (out)\n",
    "\n",
    "# parse_url_qs_for_compTag(list(公司数据选择器链结.values())[0])['compTag']\n",
    "参数模板 = parse_url_qs_for_compTag(list(公司数据选择器链结.values())[0])\n",
    "print(参数模板)\n",
    "# [ parse_url_qs_for_compTag(x)['compTag'] for x in 公司数据选择器链结.values()]\n",
    "[ parse_url_qs_for_compTag(x)['compTag'][0] for x in 公司数据选择器链结.values()]\n",
    "\n",
    "字典_compTag = { k:parse_url_qs_for_compTag(v)['compTag'][0] for k,v in 公司数据选择器链结.items()}\n",
    "print (字典_compTag)\n",
    "\n",
    "# B-5 建构 参数模板  \n",
    "def 参数模板生成(compTag , keyword ):\n",
    "    参数 = 参数模板.copy()\n",
    "    参数['compTag'] = compTag\n",
    "    参数['keyword'] = keyword\n",
    "    return (参数)\n",
    "\n",
    "参数_compTag_用户体验 = { k:参数模板生成(compTag = [v], keyword = ['用户体验']) for k,v in 字典_compTag.items()}\n",
    "print(参数_compTag_用户体验)\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>scheme</th>\n",
       "      <th>netloc</th>\n",
       "      <th>path</th>\n",
       "      <th>params</th>\n",
       "      <th>query</th>\n",
       "      <th>fragment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=260c913c0571747b&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=260c913c0571747b&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=260c913c0571747b&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=260c913c0571747b&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=260c913c0571747b&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>/zhaopin/</td>\n",
       "      <td></td>\n",
       "      <td>init=-1&amp;headckid=260c913c0571747b&amp;fromSearchBt...</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  scheme netloc       path params  \\\n",
       "0                /zhaopin/          \n",
       "1                /zhaopin/          \n",
       "2                /zhaopin/          \n",
       "3                /zhaopin/          \n",
       "4                /zhaopin/          \n",
       "5                /zhaopin/          \n",
       "\n",
       "                                               query fragment  \n",
       "0  init=-1&headckid=260c913c0571747b&fromSearchBt...           \n",
       "1  init=-1&headckid=260c913c0571747b&fromSearchBt...           \n",
       "2  init=-1&headckid=260c913c0571747b&fromSearchBt...           \n",
       "3  init=-1&headckid=260c913c0571747b&fromSearchBt...           \n",
       "4  init=-1&headckid=260c913c0571747b&fromSearchBt...           \n",
       "5  init=-1&headckid=260c913c0571747b&fromSearchBt...           "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scheme      1\n",
      "netloc      1\n",
      "path        1\n",
      "params      1\n",
      "query       5\n",
      "fragment    1\n",
      "dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>init</th>\n",
       "      <th>headckid</th>\n",
       "      <th>fromSearchBtn</th>\n",
       "      <th>keyword</th>\n",
       "      <th>ckid</th>\n",
       "      <th>siTag</th>\n",
       "      <th>d_sfrom</th>\n",
       "      <th>d_ckId</th>\n",
       "      <th>d_curPage</th>\n",
       "      <th>d_pageSize</th>\n",
       "      <th>d_headId</th>\n",
       "      <th>curPage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1</td>\n",
       "      <td>260c913c0571747b</td>\n",
       "      <td>2</td>\n",
       "      <td>PRD</td>\n",
       "      <td>260c913c0571747b°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1</td>\n",
       "      <td>260c913c0571747b</td>\n",
       "      <td>2</td>\n",
       "      <td>PRD</td>\n",
       "      <td>260c913c0571747b°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-1</td>\n",
       "      <td>260c913c0571747b</td>\n",
       "      <td>2</td>\n",
       "      <td>PRD</td>\n",
       "      <td>260c913c0571747b°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-1</td>\n",
       "      <td>260c913c0571747b</td>\n",
       "      <td>2</td>\n",
       "      <td>PRD</td>\n",
       "      <td>260c913c0571747b°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1</td>\n",
       "      <td>260c913c0571747b</td>\n",
       "      <td>2</td>\n",
       "      <td>PRD</td>\n",
       "      <td>260c913c0571747b°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>-1</td>\n",
       "      <td>260c913c0571747b</td>\n",
       "      <td>2</td>\n",
       "      <td>PRD</td>\n",
       "      <td>260c913c0571747b°radeFlag=0</td>\n",
       "      <td>1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw</td>\n",
       "      <td>search_unknown</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>917a54aacc3a11183fcafabd322be908</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  init          headckid fromSearchBtn keyword                         ckid  \\\n",
       "0   -1  260c913c0571747b             2     PRD  260c913c0571747b°radeFlag=0   \n",
       "1   -1  260c913c0571747b             2     PRD  260c913c0571747b°radeFlag=0   \n",
       "2   -1  260c913c0571747b             2     PRD  260c913c0571747b°radeFlag=0   \n",
       "3   -1  260c913c0571747b             2     PRD  260c913c0571747b°radeFlag=0   \n",
       "4   -1  260c913c0571747b             2     PRD  260c913c0571747b°radeFlag=0   \n",
       "5   -1  260c913c0571747b             2     PRD  260c913c0571747b°radeFlag=0   \n",
       "\n",
       "                                           siTag         d_sfrom  \\\n",
       "0  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "1  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "2  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "3  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "4  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "5  1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw  search_unknown   \n",
       "\n",
       "                             d_ckId d_curPage d_pageSize  \\\n",
       "0  917a54aacc3a11183fcafabd322be908         0         40   \n",
       "1  917a54aacc3a11183fcafabd322be908         0         40   \n",
       "2  917a54aacc3a11183fcafabd322be908         0         40   \n",
       "3  917a54aacc3a11183fcafabd322be908         0         40   \n",
       "4  917a54aacc3a11183fcafabd322be908         0         40   \n",
       "5  917a54aacc3a11183fcafabd322be908         0         40   \n",
       "\n",
       "                           d_headId curPage  \n",
       "0  917a54aacc3a11183fcafabd322be908       1  \n",
       "1  917a54aacc3a11183fcafabd322be908       2  \n",
       "2  917a54aacc3a11183fcafabd322be908       3  \n",
       "3  917a54aacc3a11183fcafabd322be908       4  \n",
       "4  917a54aacc3a11183fcafabd322be908       1  \n",
       "5  917a54aacc3a11183fcafabd322be908       9  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "init             1\n",
      "headckid         1\n",
      "fromSearchBtn    1\n",
      "keyword          1\n",
      "ckid             1\n",
      "siTag            1\n",
      "d_sfrom          1\n",
      "d_ckId           1\n",
      "d_curPage        1\n",
      "d_pageSize       1\n",
      "d_headId         1\n",
      "curPage          5\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# A-2 建构参数模板：找到关键参数及参数结构\n",
    "\n",
    "# 需要模组库\n",
    "from urllib.parse import urlparse, parse_qs  \n",
    "# （然）\n",
    "### python的urllib.parse库解析URL\n",
    "#### 解析url\n",
    "##### urlparse() 函数可以将 URL 解析成 ParseResult 对象。对象中包含了六个元素，分别为：\n",
    "# - 协议（scheme）\n",
    "# - 域名（netloc）\n",
    "# - 路径（path）\n",
    "# - 路径参数（params）\n",
    "# - 查询参数（query）\n",
    "# - 片段（fragment）\n",
    "\n",
    "import pandas as pd\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "# 总体目标：输入 href_列表, 建构出参数字典\n",
    "\n",
    "# urlparse 解析后丢入数据框\n",
    "df = pd.DataFrame([ urlparse(x) for x in href_列表])  ## （然）用于生成表格，使用pd.DataFrame\n",
    "df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])  ## （然）使数据对应输出？\n",
    "\n",
    "display(df)\n",
    "print(df.nunique())\n",
    "display(df_qs)\n",
    "print(df_qs.nunique())\n",
    "\n",
    "df_qs.curPage\n",
    "df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 观察：\n",
    "* query\n",
    "* curPage 5次, 最大值9, 本页不算?\n",
    "\n",
    "#### 答：\n",
    "#### （然）：只生成本页有的额外翻页URL, 并没有推估到&curPage=9,也没有这页，猜测需要从第0页开始，而不是第一页开始；然后需要+1跳转到下一页。\n",
    "-----\n",
    "\n",
    "## 建构参数模板：curPage\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'init': ['-1'], 'headckid': ['260c913c0571747b'], 'fromSearchBtn': ['2'], 'keyword': ['PRD'], 'ckid': ['260c913c0571747b°radeFlag=0'], 'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'], 'd_sfrom': ['search_unknown'], 'd_ckId': ['917a54aacc3a11183fcafabd322be908'], 'd_curPage': ['0'], 'd_pageSize': ['40'], 'd_headId': ['917a54aacc3a11183fcafabd322be908'], 'curPage': ['1']}\n",
      "{'2': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=1', '3': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=2', '4': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=3', '5': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=4', '下一页': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=1', '': '/zhaopin/?init=-1&headckid=260c913c0571747b&fromSearchBtn=2&keyword=PRD&ckid=260c913c0571747b°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=917a54aacc3a11183fcafabd322be908&d_curPage=0&d_pageSize=40&d_headId=917a54aacc3a11183fcafabd322be908&curPage=9'}\n"
     ]
    }
   ],
   "source": [
    "# A-2 建构参数模板：找到关键参数及参数结构\n",
    "\n",
    "def parse_url_qs_for_curPage (url):\n",
    "    six_parts = urlparse(url) \n",
    "    out = parse_qs(six_parts.query)\n",
    "    return (out)\n",
    "\n",
    "# 取一例做模板\n",
    "参数模板 = parse_url_qs_for_curPage(href_列表[0])\n",
    "print (参数模板)\n",
    "\n",
    "print (href_字典)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "9\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{0: {'init': ['-1'],\n",
       "  'headckid': ['260c913c0571747b'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['260c913c0571747b°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'curPage': [0]},\n",
       " 1: {'init': ['-1'],\n",
       "  'headckid': ['260c913c0571747b'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['260c913c0571747b°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'curPage': [1]},\n",
       " 2: {'init': ['-1'],\n",
       "  'headckid': ['260c913c0571747b'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['260c913c0571747b°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'curPage': [2]},\n",
       " 3: {'init': ['-1'],\n",
       "  'headckid': ['260c913c0571747b'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['260c913c0571747b°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'curPage': [3]},\n",
       " 4: {'init': ['-1'],\n",
       "  'headckid': ['260c913c0571747b'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['260c913c0571747b°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'curPage': [4]},\n",
       " 5: {'init': ['-1'],\n",
       "  'headckid': ['260c913c0571747b'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['260c913c0571747b°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'curPage': [5]},\n",
       " 6: {'init': ['-1'],\n",
       "  'headckid': ['260c913c0571747b'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['260c913c0571747b°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'curPage': [6]},\n",
       " 7: {'init': ['-1'],\n",
       "  'headckid': ['260c913c0571747b'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['260c913c0571747b°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'curPage': [7]},\n",
       " 8: {'init': ['-1'],\n",
       "  'headckid': ['260c913c0571747b'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['260c913c0571747b°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'curPage': [8]},\n",
       " 9: {'init': ['-1'],\n",
       "  'headckid': ['260c913c0571747b'],\n",
       "  'fromSearchBtn': ['2'],\n",
       "  'keyword': ['用户体验'],\n",
       "  'ckid': ['260c913c0571747b°radeFlag=0'],\n",
       "  'siTag': ['1B2M2Y8AsgTpgAmY7PhCfg~fA9rXquZc5IkJpXC-Ycixw'],\n",
       "  'd_sfrom': ['search_unknown'],\n",
       "  'd_ckId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'd_curPage': ['0'],\n",
       "  'd_pageSize': ['40'],\n",
       "  'd_headId': ['917a54aacc3a11183fcafabd322be908'],\n",
       "  'curPage': [9]}}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A-3 建构参数模板生成器：keyword curPage\n",
    "def 参数模板生成(keyword, curPage):\n",
    "    参数 = 参数模板.copy()\n",
    "    参数['curPage'] = curPage\n",
    "    参数['keyword'] = keyword\n",
    "    return (参数)\n",
    "\n",
    "参数_keyword_用户体验_curPage = { \n",
    "    i:参数模板生成(curPage = [i], \\\n",
    "                  keyword = ['用户体验']) \\\n",
    "    for i,v in href_字典.items()\\\n",
    "    }\n",
    "\n",
    "# print(参数_keyword_用户体验_curPage) # 只生成本页有的额外翻页URL, 并没有推估到&curPage=9,也没有这页\n",
    "\n",
    "print (df_qs.curPage_int.min()) # 最小值只有1\n",
    "print (df_qs.curPage_int.max()) # 最大值只有9\n",
    "\n",
    "### （然）上面两行的代码是用算的，并不是给定值\n",
    "\n",
    "# 应该是 0 (本页)....9(最大值)\n",
    "\n",
    "参数_keyword_用户体验_curPage = { \n",
    "    i:参数模板生成(curPage = [i], \\\n",
    "                  keyword = ['用户体验']) \\\n",
    "    for i in range(0,df_qs.curPage_int.max()+1)\\   # （然）需要通过加一来请求下一页内容\n",
    "    }\n",
    "参数_keyword_用户体验_curPage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：系统性迭代\n",
    "\n",
    "## 爬亦有道\n",
    "* robots.txt 站长/网站拥有者给搜索引擎的\"道\"\n",
    "* 频率及时间\n",
    "  * 不要爬太快\n",
    "  * 尽量像\"人\"一样礼貌\n",
    "  * time.sleep  \n",
    "  <mark>（然）</mark>设置停顿的描述，让我们爬取时更像是“人”在操作，不要爬取太快，造成对网站的伤害\n",
    "  \n",
    "```python\n",
    "\n",
    "# 上周C-3   多个页面\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "\n",
    "list_df = list()\n",
    "for k,v in 参数_compTag_用户体验.items():\n",
    "    payload = v\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    df = df.assign (热门公司类型 = k)    \n",
    "    list_df.append(df)\n",
    "\n",
    "df_all = pd.concat(list_df)\n",
    "df_all\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# B-1 上周C-2  简化 上上周A-1   单一页面爬+解析\n",
    "session = HTMLSession()\n",
    "\n",
    "def requests_liepin( url, params):\n",
    "    r = session.get( url , params = payload)\n",
    "\n",
    "    # 先取特定元素, 精准打击其子后辈\n",
    "    主要元素 = r.html.xpath( '//ul[@class=\"sojob-list\"]/li')\n",
    "\n",
    "    # 作为xpath字典，键为我要抓的牛肉名称，值为xpath\n",
    "    dict_xpaths={ \n",
    "        'text': {\n",
    "            'edu':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]',  ## （然）找到学历的xpath\n",
    "            '经验':      '//div[contains(@class,\"job-info\")]/p/span[@class=\"edu\"]/following-sibling::span', ## （然）找到经验的xpath\n",
    "            '薪水':    '//div[contains(@class,\"job-info\")]/p/span[@class=\"text-warning\"]', ## （然）找到薪水的xpath\n",
    "            '时间':    '//div[contains(@class,\"job-info\")]/p/time/@title',   ## （然）找到时间的xpath\n",
    "            '职称':    '//div[contains(@class,\"job-info\")]/h3/a',          ## （然）找到职位名称的xpath\n",
    "            '公司地点': '//div[contains(@class,\"job-info\")]/p/a',       ## （然）找到公司地点的xpath\n",
    "            '公司名称': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a',  ## （然）找到公司名称的xpath\n",
    "        },\n",
    "        'text_content': {\n",
    "        },\n",
    "        'href': {\n",
    "            '链结':    '//div[contains(@class,\"job-info\")]/h3/a',   ##  （然）找到链接的xpath\n",
    "            '公司URL': '//div[contains(@class,\"sojob-item-main\")]//p[@class=\"company-name\"]/a', #（然）找到公司url的xpath\n",
    "        }\n",
    "    }\n",
    "\n",
    "    def get_e_text_content(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [e.xpath(_xpath_)[0].lxml.text_content() for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_text(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [\"\".join([x.strip() if type(x) is str else x.text.strip() for x in e.xpath(_xpath_)]) for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    def get_e_href(_xpath_):\n",
    "        # 高级列表推导\n",
    "        暂存结果 = [list(e.xpath(_xpath_, first=True).absolute_links)[0] \\\n",
    "                   if len(e.xpath(_xpath_, first=True).absolute_links) >= 1  \\\n",
    "                   else \"\" for e in 主要元素]\n",
    "        return(暂存结果)\n",
    "\n",
    "    # 只对主要元素下进行.xpath取值\n",
    "    数据字典 = dict()\n",
    "\n",
    "    数据字典 = {k:get_e_text_content(v) for k,v in dict_xpaths['text_content'].items()}  ## （然）for循环遍历，循环text_content\n",
    "    数据字典.update({k:get_e_text(v) for k,v in dict_xpaths['text'].items()})   ## (然) for循环遍历，循环抓取text列表里面的内容，存储在新字典里\n",
    "    数据字典.update({k:get_e_href(v) for k,v in dict_xpaths['href'].items()})   ## （然）for循环，循环抓取href列表里面的内容，存储在新字典\n",
    "\n",
    "    数据 = pd.DataFrame(数据字典)  # （然）使用pd.DataFrame生成数据表格\n",
    "    #数据.to_excel(\"20春_Web数据挖掘_week03_liepin.xlsx\", sheet_name=\"搜查结果\")\n",
    "    return (数据)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 爬亦有道- 不要爬太快\n",
    "time.sleep\n",
    "<mark>(然）</mark>\n",
    "- Python time sleep() 函数推迟调用线程的运行，可通过参数secs指秒数，表示进程挂起的时间。\n",
    "- 语法 time.sleep(t)\n",
    "- 参数 t -- 推迟执行的秒数。\n",
    "- 返回值：该函数没有返回值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 4.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "time.sleep(3+4*random())   # （然）random（）表示0-1的随机数，所以该行代码表示停顿3-7秒的随机数，平均5秒 如：3+4*0=3   3+4*1=7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>edu</th>\n",
       "      <th>经验</th>\n",
       "      <th>薪水</th>\n",
       "      <th>时间</th>\n",
       "      <th>职称</th>\n",
       "      <th>公司地点</th>\n",
       "      <th>公司名称</th>\n",
       "      <th>链结</th>\n",
       "      <th>公司URL</th>\n",
       "      <th>curPage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>本科及以上</td>\n",
       "      <td>7年以上</td>\n",
       "      <td>面议</td>\n",
       "      <td>2020年04月04日</td>\n",
       "      <td>营销副总（项目）</td>\n",
       "      <td>佛山</td>\n",
       "      <td>中国电建地产集团华南区域总部</td>\n",
       "      <td>https://www.liepin.com/job/1927229275.shtml</td>\n",
       "      <td>https://www.liepin.com/company/9624513/</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>本科及以上</td>\n",
       "      <td>7年以上</td>\n",
       "      <td>面议</td>\n",
       "      <td>2020年04月04日</td>\n",
       "      <td>营销副总（项目）</td>\n",
       "      <td>广州-石井</td>\n",
       "      <td>中国电建地产集团华南区域总部</td>\n",
       "      <td>https://www.liepin.com/job/1927229273.shtml</td>\n",
       "      <td>https://www.liepin.com/company/9624513/</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>本科及以上</td>\n",
       "      <td>7年以上</td>\n",
       "      <td>面议</td>\n",
       "      <td>2020年04月04日</td>\n",
       "      <td>营销副总（项目）</td>\n",
       "      <td>深圳</td>\n",
       "      <td>中国电建地产集团华南区域总部</td>\n",
       "      <td>https://www.liepin.com/job/1927229271.shtml</td>\n",
       "      <td>https://www.liepin.com/company/9624513/</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>本科及以上</td>\n",
       "      <td>1年以上</td>\n",
       "      <td>10-12k·13薪</td>\n",
       "      <td>2020年04月04日</td>\n",
       "      <td>c++开发工程师</td>\n",
       "      <td>北京</td>\n",
       "      <td>北京赛普泰克技术有限公司</td>\n",
       "      <td>https://www.liepin.com/job/1927229255.shtml</td>\n",
       "      <td>https://www.liepin.com/company/7884122/</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>统招本科</td>\n",
       "      <td>6年以上</td>\n",
       "      <td>15-25k·16薪</td>\n",
       "      <td>2020年04月04日</td>\n",
       "      <td>干部管理经理</td>\n",
       "      <td>深圳</td>\n",
       "      <td>华星光电</td>\n",
       "      <td>https://www.liepin.com/job/1927228767.shtml</td>\n",
       "      <td>https://www.liepin.com/company/671058/</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>大专及以上</td>\n",
       "      <td>2年以上</td>\n",
       "      <td>6-10k·12薪</td>\n",
       "      <td>2020年04月03日</td>\n",
       "      <td>人力资源专员/招聘</td>\n",
       "      <td>宁波-鄞州区</td>\n",
       "      <td>宁波渠成集团有限公司</td>\n",
       "      <td>https://www.liepin.com/job/1925226785.shtml</td>\n",
       "      <td>https://www.liepin.com/company/8096424/</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>统招本科</td>\n",
       "      <td>经验不限</td>\n",
       "      <td>8-15k·14薪</td>\n",
       "      <td>2020年04月03日</td>\n",
       "      <td>卫星电源控制器研发工程师</td>\n",
       "      <td>苏州-虎丘区</td>\n",
       "      <td>苏州馥昶空间技术有限公司</td>\n",
       "      <td>https://www.liepin.com/job/1925224865.shtml</td>\n",
       "      <td>https://www.liepin.com/company/9665557/</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>本科及以上</td>\n",
       "      <td>3年以上</td>\n",
       "      <td>面议</td>\n",
       "      <td>2020年04月03日</td>\n",
       "      <td>销售部销售经理</td>\n",
       "      <td>上海</td>\n",
       "      <td>上海大丁自动化科技有限公司</td>\n",
       "      <td>https://www.liepin.com/job/1925196603.shtml</td>\n",
       "      <td>https://www.liepin.com/company/10058443/</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>大专及以上</td>\n",
       "      <td>2年以上</td>\n",
       "      <td>5-8k·12薪</td>\n",
       "      <td>2020年04月03日</td>\n",
       "      <td>ie工程师</td>\n",
       "      <td>中山</td>\n",
       "      <td>中山市皇鼎五金制品有限公司</td>\n",
       "      <td>https://www.liepin.com/job/1925097067.shtml</td>\n",
       "      <td>https://www.liepin.com/company/9190284/</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>本科及以上</td>\n",
       "      <td>3年以上</td>\n",
       "      <td>7-20k·12薪</td>\n",
       "      <td>2020年04月03日</td>\n",
       "      <td>广告销售经理</td>\n",
       "      <td>上海-长宁区</td>\n",
       "      <td>上海魄力广告传媒有限公司</td>\n",
       "      <td>https://www.liepin.com/job/1925095091.shtml</td>\n",
       "      <td>https://www.liepin.com/company/8239390/</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>400 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      edu    经验          薪水           时间            职称    公司地点  \\\n",
       "0   本科及以上  7年以上          面议  2020年04月04日      营销副总（项目）      佛山   \n",
       "1   本科及以上  7年以上          面议  2020年04月04日      营销副总（项目）   广州-石井   \n",
       "2   本科及以上  7年以上          面议  2020年04月04日      营销副总（项目）      深圳   \n",
       "3   本科及以上  1年以上  10-12k·13薪  2020年04月04日      c++开发工程师      北京   \n",
       "4    统招本科  6年以上  15-25k·16薪  2020年04月04日        干部管理经理      深圳   \n",
       "..    ...   ...         ...          ...           ...     ...   \n",
       "35  大专及以上  2年以上   6-10k·12薪  2020年04月03日     人力资源专员/招聘  宁波-鄞州区   \n",
       "36   统招本科  经验不限   8-15k·14薪  2020年04月03日  卫星电源控制器研发工程师  苏州-虎丘区   \n",
       "37  本科及以上  3年以上          面议  2020年04月03日       销售部销售经理      上海   \n",
       "38  大专及以上  2年以上    5-8k·12薪  2020年04月03日         ie工程师      中山   \n",
       "39  本科及以上  3年以上   7-20k·12薪  2020年04月03日        广告销售经理  上海-长宁区   \n",
       "\n",
       "              公司名称                                           链结  \\\n",
       "0   中国电建地产集团华南区域总部  https://www.liepin.com/job/1927229275.shtml   \n",
       "1   中国电建地产集团华南区域总部  https://www.liepin.com/job/1927229273.shtml   \n",
       "2   中国电建地产集团华南区域总部  https://www.liepin.com/job/1927229271.shtml   \n",
       "3     北京赛普泰克技术有限公司  https://www.liepin.com/job/1927229255.shtml   \n",
       "4             华星光电  https://www.liepin.com/job/1927228767.shtml   \n",
       "..             ...                                          ...   \n",
       "35      宁波渠成集团有限公司  https://www.liepin.com/job/1925226785.shtml   \n",
       "36    苏州馥昶空间技术有限公司  https://www.liepin.com/job/1925224865.shtml   \n",
       "37   上海大丁自动化科技有限公司  https://www.liepin.com/job/1925196603.shtml   \n",
       "38   中山市皇鼎五金制品有限公司  https://www.liepin.com/job/1925097067.shtml   \n",
       "39    上海魄力广告传媒有限公司  https://www.liepin.com/job/1925095091.shtml   \n",
       "\n",
       "                                       公司URL  curPage  \n",
       "0    https://www.liepin.com/company/9624513/        0  \n",
       "1    https://www.liepin.com/company/9624513/        0  \n",
       "2    https://www.liepin.com/company/9624513/        0  \n",
       "3    https://www.liepin.com/company/7884122/        0  \n",
       "4     https://www.liepin.com/company/671058/        0  \n",
       "..                                       ...      ...  \n",
       "35   https://www.liepin.com/company/8096424/        9  \n",
       "36   https://www.liepin.com/company/9665557/        9  \n",
       "37  https://www.liepin.com/company/10058443/        9  \n",
       "38   https://www.liepin.com/company/9190284/        9  \n",
       "39   https://www.liepin.com/company/8239390/        9  \n",
       "\n",
       "[400 rows x 10 columns]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# B-2 多个页面，但放慢脚步 time.sleep\n",
    "%%time\n",
    "\n",
    "import time  ## （然）引用时间模块\n",
    "from random import random  ## （然）引用random的随机模块\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "\n",
    "list_df = list()\n",
    "for k,v in 参数_keyword_用户体验_curPage.items():\n",
    "    payload = v\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    time.sleep(3+4*random())  # 放慢脚步 3-7秒, 平均约5秒\n",
    "    df = df.assign (curPage = k)  # 区分  curPage #（然）用于记录第几页，存储数据部分\n",
    "    list_df.append(df)\n",
    "\n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "# 上周C-4   输出\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页.xlsx\",\\    ## 使用df将数据存入表格，以表格形式展示\n",
    "                sheet_name=\"用户体验\")\n",
    "\n",
    "# 预估时间: 5秒*10 =50\n",
    "# 预估数量: 40*10 =400"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 多个页面+多个关键词\n",
    "time.sleep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "用户体验 10\n",
      "UX 10\n",
      "Wall time: 1min 54s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# B-3 多个页面+多个关键词\n",
    "import time    # （然）引用时间模块\n",
    "from random import random  # （然）引用random模块\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'   # （然）找到翻页的xpath\n",
    "\n",
    "keywords = ['用户体验','UX']  ## （然）多个关键词\n",
    "list_df = list()\n",
    "\n",
    "## 第一页试探有多长的页面\n",
    "for key in keywords:\n",
    "    payload = 参数模板生成(keyword=[key], curPage=['0'])\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "    df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "    df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "    df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数\n",
    "    长度 = df_qs.curPage_int.max()+1    ## （然）通过加1跳转至下一页\n",
    "    参数_keyword_X_curPage = { \n",
    "        i:参数模板生成(curPage = [i], \\\n",
    "                      keyword = [key]) \\\n",
    "        for i in range(0,长度)\\\n",
    "        }\n",
    "    #print (参数_keyword_X_curPage)\n",
    "    print (key,长度)\n",
    "    \n",
    "    for k,v in 参数_keyword_X_curPage.items():\n",
    "        payload = v\n",
    "        df = requests_liepin( url, params = payload)\n",
    "        time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "        df = df.assign (keyword = key)  # 区分  keyword     # （然）与下一行都为存储数据部分\n",
    "        df = df.assign (curPage = k)  # 区分  curPage    #（然）记录第几页\n",
    "        list_df.append(df)\n",
    "        \n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页.xlsx\",\\   ## （然）输出表格，存储数据\n",
    "                sheet_name=\"_\".join(keywords))\n",
    "# 预估时间: 2*5秒*10 =100\n",
    "# 预估数量: 2*40*10 =800"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 翻页：数据备份与整合\n",
    "多个页面+多个关键词执行时，若怕中断最好把每一页的df内容备份做中继"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "用户体验 10\n",
      "UX 10\n",
      "产品需求 10\n",
      "PRD 10\n",
      "Wall time: 3min 54s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# C-1 多个页面+多个关键词\n",
    "import time   ##(然) 使用时间模块\n",
    "from random import random  ##(然)使用random随机模块\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/\"   # （然）将招聘网的链接赋值给url\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'  ## （然）找到翻页的xpath,属性@href，/zhaopin的都保留\n",
    "\n",
    "keywords = ['用户体验','UX','产品需求','PRD']   ## （然）多个关键词\n",
    "list_df = list()\n",
    "\n",
    "## 第一页试探有多长的页面\n",
    "for key in keywords:      ### （然） for循环，循环抓取多个关键词 \n",
    "    payload = 参数模板生成(keyword=[key], curPage=['0'])\n",
    "    df = requests_liepin( url, params = payload)\n",
    "    href_列表 = [x.xpath('//@href')[0] for x in r.html.xpath(xpath_翻页a)]\n",
    "    df = pd.DataFrame([ urlparse(x) for x in href_列表])\n",
    "    df_qs = pd.DataFrame([{k:v[0] for k,v in parse_qs(x).items()} for x in df['query'] ])\n",
    "    df_qs = df_qs.assign (curPage_int=df_qs.curPage.astype(int)) # 变成整数\n",
    "    长度 = df_qs.curPage_int.max()+1      ##（然）加1跳转下一页\n",
    "    参数_keyword_X_curPage = { \n",
    "        i:参数模板生成(curPage = [i], \\\n",
    "                      keyword = [key]) \\\n",
    "        for i in range(0,长度)\\\n",
    "        }\n",
    "    #print (参数_keyword_X_curPage)\n",
    "    print (key,长度)\n",
    "    \n",
    "    for k,v in 参数_keyword_X_curPage.items():\n",
    "        payload = v\n",
    "        df = requests_liepin( url, params = payload)\n",
    "        time.sleep(3+4*random())  #放慢脚步 3-7秒, 平均约5秒\n",
    "        ## 备份\n",
    "        df.to_csv(\"20春_Web数据挖掘_week04_liepin_{key}_{k}.tsv\"\\\n",
    "                  .format(key=key, k=k), sep=\"\\t\", encoding=\"utf8\")   \n",
    "\n",
    "## （然）{key}{k}表示先占了两个位置，通过.format(key=key,k=k)再传入相对应的位置，encoding=“utf8”是代码类型转换，防止出现乱码。\n",
    "        \n",
    "        df = df.assign (keyword = key)  # 区分  keyword    \n",
    "        df = df.assign (curPage = k)  # 区分  curPage    \n",
    "        list_df.append(df)\n",
    "        \n",
    "df_all = pd.concat(list_df).reset_index()\n",
    "df_all.index.name = '序'\n",
    "\n",
    "df_all.to_excel(\"20春_Web数据挖掘_week04_liepin_翻页_4.xlsx\",\\   ##（然）将数据存入Excel表格\n",
    "                sheet_name=\"_\".join(keywords))\n",
    "# 预估时间: 4*5秒*10 =200\n",
    "# 预估数量: 4*40*10 =1600"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 本周练习\n",
    "\n",
    "* 开始试验各类参数的调整\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 代码记录练习区"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "### 豆瓣图书TOP(然)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://book.douban.com/top250?start=225'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### 不同链接的对比\n",
    "\n",
    "\"https://book.douban.com/top250?start=0\"   # 第一页\n",
    "\"https://book.douban.com/top250?start=25\"  # 第二页\n",
    "\"https://book.douban.com/top250?start=50\"  # 第三页\n",
    "\"https://book.douban.com/top250?start=75\"  # 第四页\n",
    "\"https://book.douban.com/top250?start=100\" # 第五页\n",
    "...\n",
    "\"https://book.douban.com/top250?start=225\" # 第十页\n",
    "\n",
    "## 可以发现每一页的链接区别在最后的start=XXX，每一页之间的链接最后值都是相隔25增加"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 58同城 keyword为ux"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element 'a' class=('current',) herf='javascript:void(0)'>, <Element 'a' href='https://gz.58.com/job/pn2/?keyword=ux'>, <Element 'a' href='https://gz.58.com/job/pn3/?keyword=ux'>, <Element 'a' href='https://gz.58.com/job/pn4/?keyword=ux'>, <Element 'a' href='https://gz.58.com/job/pn5/?keyword=ux'>, <Element 'a' href='https://gz.58.com/job/pn6/?keyword=ux'>, <Element 'a' href='https://gz.58.com/job/pn7/?keyword=ux'>, <Element 'a' href='https://gz.58.com/job/pn8/?keyword=ux'>, <Element 'a' class=('num_sure',) href=''>]\n"
     ]
    }
   ],
   "source": [
    "# 58同城招聘网跳转链接\n",
    "# 关键词keyword=ux\n",
    "import requests_html   ## （然）引用requests模块\n",
    "from requests_html import HTMLSession  ## 引用设立session值\n",
    "\n",
    "url = \"https://gz.58.com/job/?keyword=ux\"   ### （然）将页面链接赋值给url,改成了58同城的招聘网进行试验\n",
    "session = HTMLSession()                          ### （然）设立session，解析数据\n",
    "r = session.get( url )  ###（然） 获取页面信息\n",
    "xpath_翻页a = '//div[@class=\"pagesout\"]/span/a'## （然）找到翻页的xpath \n",
    "print (r.html.xpath(xpath_翻页a))     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 猎聘招聘网，keyword为ux"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element 'a' class=('first', 'disabled') href='javascript:;' title='首页'>, <Element 'a' class=('disabled',) href='javascript:;'>, <Element 'a' class=('current',) href='javascript:;'>, <Element 'a' href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=1'>, <Element 'a' href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=2'>, <Element 'a' href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=3'>, <Element 'a' href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=4'>, <Element 'a' href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=1'>, <Element 'a' class=('last',) href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=9' title='末页'>]\n"
     ]
    }
   ],
   "source": [
    "import requests_html   ## （然）引用requests模块\n",
    "from requests_html import HTMLSession  ## 引用设立session值\n",
    "\n",
    "url = \"https://www.liepin.com/zhaopin/?keyword=ux\"   ### （然）将页面链接赋值给url,尝试将keyword改成ux，进行试验\n",
    "session = HTMLSession()                          ### （然）设立session\n",
    "r = session.get( url )  ###（然） 获取页面信息\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a'## （然）找到翻页的xpath\n",
    "print (r.html.xpath(xpath_翻页a))  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element 'a' href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=1'>, <Element 'a' href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=2'>, <Element 'a' href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=3'>, <Element 'a' href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=4'>, <Element 'a' href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=1'>, <Element 'a' class=('last',) href='/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=9' title='末页'>]\n",
      "{'2': '/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=1', '3': '/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=2', '4': '/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=3', '5': '/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=4', '下一页': '/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=1', '': '/zhaopin/?init=-1&headckid=7b157dc28ab1ddeb&fromSearchBtn=2&keyword=ux&ckid=7b157dc28ab1ddeb°radeFlag=0&siTag=1B2M2Y8AsgTpgAmY7PhCfg%7EfA9rXquZc5IkJpXC-Ycixw&d_sfrom=search_unknown&d_ckId=f3bc79b1218ca1a9ac58fb2ea8f2db07&d_curPage=0&d_pageSize=40&d_headId=f3bc79b1218ca1a9ac58fb2ea8f2db07&curPage=9'}\n"
     ]
    }
   ],
   "source": [
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a' # 有disabled, current等href是javascript   ## （然）先找到翻页的xpath\n",
    "xpath_翻页a = '//div[@class=\"pagerbar\"]/a[starts-with(@href,\"/zhaopin\")]'     ## (然) 判别式保留，@href是属性，以/zhaopin结尾的都要\n",
    "print (r.html.xpath(xpath_翻页a)) # 物件   ## （然）将获取的结果打印出来\n",
    "\n",
    "链接_列表 = [s.xpath('//@href')[0] for s in r.html.xpath(xpath_翻页a)]  ## （然）获取第一个列表元素，设置了for循环\n",
    "#print (href_列表)\n",
    "\n",
    "文字_列表 = [s.text for s in r.html.xpath(xpath_翻页a)]\n",
    "#print (文字_列表)\n",
    "\n",
    "链接_字典 = {s.text:s.xpath('//@href')[0]  for s in r.html.xpath(xpath_翻页a)}\n",
    "print (链接_字典)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### web数据挖掘类型（三类）（然）\n",
    "- 最常见的是对于网站内容的爬取，包括文本、图片和文件等；\n",
    "- 其次是对于网站结构的爬取，包括网站目录，链接之间的相互跳转关系，二级域名等；\n",
    "- 还有一种爬虫是对于Web应用数据的挖掘，包括获取网站CMS类型，Web插件等。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  爬虫伦理（然）\n",
    "#### 爬亦有道\n",
    "- 不要爬太快，要像“人”一样礼貌\n",
    "- 就像是两个人在来来往往的相处中，会考虑对方的感受；在互联网的世界中，我们也要考虑一下服务器对爬虫的感受是怎样的。\n",
    "- 服务器其实就是一个超级电脑，拥有这个服务器的公司，对爬虫其实也有明确的态度。\n",
    "- 通常情况下，服务器不太会在意小爬虫，但是，服务器会拒绝频率很高的大型爬虫和恶意爬虫，因为这会给服务器带来极大的压力或伤害。\n",
    "- robots协议是互联网爬虫的一项公认的道德规范，它的全称是“网络爬虫排除标准”（robots exclusion protocol），这个协议用来告诉爬虫，哪些页面是可以抓取的，哪些不可以。\n",
    "- robots协议是“分段”的，一种是User-agent:，另一种是Allow:或Disallow:\n",
    "-----------------------------------------------------------------------------------------------------------------------------------------\n",
    "- User-agent表示的是爬虫类型\n",
    "- Allow代表允许被访问，Disallow代表禁止被访问。\n",
    "- 互联网并非法外之地，和爬虫相关的法律也在建立和完善之中，目前通用的伦理规范就是robots协议，我们在爬取网络中的信息时，应该有意识地去遵守这个协议。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 不足之处"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 对于参数的理解还是有欠缺\n",
    "- 当试验别的网站时无法操作成功出表格，只能简单地成功打印出它的页面跳转链接"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 笔记链接"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [note1](https://gitee.com/neria/web-mining/blob/master/week01-note.md)\n",
    "- [note2](https://gitee.com/neria/web-mining/blob/master/week02-note.md)\n",
    "- [note3](https://gitee.com/neria/web-mining/blob/master/week02%E5%B0%8F%E7%BB%93.md)\n",
    "- [note4](https://gitee.com/neria/web-mining/blob/master/week03-note.md)\n",
    "- [改动说明文档](https://gitee.com/neria/web-mining/blob/master/week04.md)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "749px",
    "left": "1125.609375px",
    "top": "110px",
    "width": "281.390625px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
